Learning Objects and Virtual Learning Environments 
Technical Evaluation Criteria 


Eugenijus Kurilovas and Valentina Dagiene 

Institute of Mathematics and Informatics, Vilnius, Lithuania 

Eugeniius.Kurilovas@itc.smm.lt 

Daqiene@ktl.mii.lt 

Abstract: The main scientific problems investigated in this article deal with technical evaluation of quality 
attributes of the main components of e-Learning systems (referred here as DLEs - Digital Libraries of 
Educational Resources and Services), i.e. , Learning Objects (LOs) and Virtual Learning Environments (VLEs). 
The main research object of the work is the effectiveness of methods of DLE components quality evaluation. The 
aim of the article is to analyse popular existing LO and VLE technical evaluation tools, and to formulate new more 
complex tools for technical quality evaluation of LOs and VLEs based on requirements for flexible DLE, as well as 
to evaluate most popular open source VLEs against new more complex criteria. Complex tools have been 
created for the evaluation of DLE components, based on a flexible approach. The authors have analysed existing 
tools for technical evaluation of LOs, and it was investigated that these tools have a number of limitations. Some 
of these tools do not examine different LO life cycle stages, and other insufficiently examine technical evaluation 
criteria before LO inclusion in the repository. All these tools insufficiently examine LOs reusability criteria. 
Therefore more complex LO technical evaluation tool is needed. It was investigated that this new more complex 
LO technical evaluation tool should include LO technical evaluation criteria suitable for different LO life cycle 
stages, including criteria before, during and after LO inclusion in the repository as well as LO reusability criteria. 
The authors have also examined several VLE technical evaluation tools suitable for flexible DLE, and it was 
investigated that these tools have a number of limitations. Several tools practically do not examine VLE 
adaptation capabilities criteria, and the other insufficiently examines general technical criteria. More complex VLE 
technical evaluation tool is needed. Therefore the authors have proposed an original more complex set of VLE 
technical evaluation criteria combining (1) General (Overall architecture and implementation; Interoperability; 
Internationalisation and Localisation; Accessibility) and (2) Adaptation (Adaptability; Personalisation; Extensibility 
and Adaptivity) VLE technical evaluation criteria. The authors have also selected and proposed to use the 
universal, clear and convenient DLE components’ evaluation rating tool, and have evaluated three most popular 
open source VLEs against technical (both general and adaptation) criteria in conformity with this rating tool. 

Keywords: managing quality in e-learning, technical evaluation, virtual learning environments, learning objects, 
repositories 

1. Technical evaluation of learning objects 

1 .1 Different approaches to learning objects technical evaluation 

The various approaches to LOs attempt to meet two common objectives: 

■ To reduce the overall costs of LOs. 

■ To obtain better LOs (Wiley 2003). 

It can be argued that the provision of LOs provides better access to quality LOs and supports 
enhanced learning outcomes. The purpose of LOs is to increase the effectiveness of learning by 
making content more readily available, by reducing the cost and effort to produce quality content, and 
by allowing content to be more easily shared. These two purposes, effectiveness and efficiency, 
receive differing emphases from different sectors (Haughey and Muirhead 2005). 

The evaluation of LOs is a comparatively new concern as the quantity of LOs has grown and the 
development of LO repositories has come about to allow for greater ease in finding and using LOs for 
both classroom and online instruction. The growth in the number of LOs, the multiplicity of authors, 
their increasing diversity of design and their availability to trained and untrained educators has 
generated interest in how to evaluate them and which criteria to use to make judgments about their 
quality and usefulness (Haughey and Muirhead 2005). 
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1.2 LORI quality criteria 

The need to evaluate LOs requires the development of criteria to be used in judging them. (Vargo et 
al. 2003) developed a Learning Object Review Instrument or LORI to evaluate LOs. The LORI 
approach uses the following ten criteria when examining LOs: 

■ Presentation: Aesthetics. 

■ Presentation: Design for learning. 

■ Accuracy of content. 

■ Support for learning goals. 

■ Motivation. 

■ Interaction: Usability. 

■ Interaction: Feedback and adaptation. 

■ Reusability (Technical criterion - authors’ comment). 

■ Metadata and interoperability compliance (Technical criterion - authors’ comment). 

■ Accessibility (Technical criterion - authors’ comment). 

The criteria were drawn from a review of pertinent literature on instructional design, computer science, 
multimedia development and educational psychology. Each measure was weighted equally and was 
rated on a four point scale from “weak” to “moderate” to “strong” to “perfect”. The LORI process 
involved both individual and group rating of LOs (Vargo et al. 2003). 

1.3 Paulsson and Naeve quality criteria 

Six action areas for establishing LO technical quality criteria are suggested by Paulsson and Naeve 
(2006): 

■ A narrow definition. 

■ A mapping taxonomy. 

■ More extensive standards. 

■ Best practise for use of existing standards. 

■ Architecture models. 

■ The separation of pedagogy from the supporting technology of LOs. 

Most LO implementations do not by far meet this vision. For those reasons it is essential to establish 
common criteria of quality for LOs. Technical quality criteria are specific characteristics and properties 
that LOs must (or in some cases ought to) adhere to - including best practice, guidelines and 
standard specifications - in order to be regarded as LOs. 

The focus in (Paulsson and Naeve 2006) is on technical quality criteria for LOs. Other quality criteria, 
such as pedagogical quality, usability or functional quality are out of scope. Such aspects of quality 
are addressed by Van Assche and Vourikari (2006), where they suggest a quality framework for the 
whole life cycle of LOs. 

The (Paulsson and Naeve 2006) evaluation focused on: (1) architecture - in terms of separation of 
data, logics, presentation, and implementation of interaction interfaces; (2) pedagogical 
contextualisation; (3) the use of standards and the extent to which they are 
decomposable/composable. Many of those issues are directly or indirectly related to the lack of 
explicit definitions and clear architectural models, together with technical (as well as other) quality 
criteria that are directly related to technical architecture. Many of the pedagogical dependencies and 
shortcomings seem to be caused by technical bindings of content to presentation and application 
logics as well as built in instructional design elements. 

The (Paulsson and Naeve 2006) study has shown that there is a huge discrepancy between different 
definitions of the LO concept. This makes it hard (if not impossible) to author LOs that have the 
qualities that LOs are often ascribed in terms of reusability, interoperability, and context 
independence. Definitions really range from “anything to everything” (McGreal 2004). However, the 
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real problem lies in that there is no separation of “anything to everything” from a technological 
perspective and “anything to everything” from a “content” perspective. “Anything to everything” from a 
“content” perspective is a good thing as this makes it easier to support different pedagogical 
directions and methods, but “anything to everything” from a technological perspective becomes 
unmanageable. Paulsson and Naeve (2006) suggest the technical and pedagogical definitions of LOs 
to be separated - within a common definition of LOs. 

The lack of common low-level definitions and models is a threat to interoperability, technical quality as 
well as for the acceptance of the LO concept itself. The Paulsson and Naeve (2006) study shows that 
the pedagogical content is often of good quality and that the ambitions are set high, but that LOs still 
do not live up to the expectations that would make them context independent, reusable objects. One 
important reason is that little consideration is given to fundamental software design principles, such as 
layering, principles from object orientation, structuring of data etc., which could enhance such 
properties that are usually ascribed to LOs. As most implementations do not deliver what they 
promise, the vision has yet to be fulfilled. There is a need to move on from just describing properties 
and characteristics, to determine how those can be realised. 

To address the identified problems Paulsson and Naeve (2006) suggest six areas for action in order 
to establish technical quality criteria for LOs: 

■ There is a need for a common (more narrow) definition of what is, and what is not a LO. 

■ In connection to narrowing down the definitions, there is a need for a taxonomy that maps on to 
the definition and where granularities as well as special properties are regarded. 

■ Standards used for LOs should be extended to go beyond descriptive information, such as 
metadata, sequencing, and packaging to also embrace standards for interfaces, “machine 
readable” descriptions of technical properties and interaction interfaces. 

■ There is a need to establish standards and recommendations that address the internal use of 
data formats and data structure. Such general technology standards exist, but seem to be rarely 
used in the LO community. 

■ It should be prescribed for the architecture of LOs to be layered as a part of best practise, in order 
to separate data, presentation and application logics. This would enhance the level of 
decomposability and context independence, as is also pointed out by Pinkwart et al. (2005). 
Layering (or multi-tier architectures) is used frequently in many other areas of application/system 
development for the very same reasons. 

■ Pedagogy should preferably be kept outside the LO in order to facilitate pedagogical context 
independence. It is suggested that the pedagogical model is added as LOs are assembled to form 
learning modules. Using such methodology, it becomes possible to do pedagogical 
contextualisation at a later stage in the authoring process, and enhance reusability of different 
components as well as components mutual pedagogical context independence. In some cases 
there might be a need to add such “instructional properties” inside LOs, but in such cases this 
should be handled in a separate layer, using standard specifications for that purpose, and not by 
hard coded implementations (Paulsson and Naeve 2006). 

1.4 MELT project quality criteria 

The MELT content audit included an in-depth examination of project partners’ existing content quality 
guidelines and produced a checklist to help them decide what content from their repositories should 
be made available in the project for enrichment. This checklist is divided into five categories - 
pedagogical, usability, reusability, accessibility and production (MELT 2007). 

The list is by no means prescriptive and not all of the criteria can always be applied to all LOs. For 
example, some LOs may score strongly in terms of reusability because they include open source 
code that facilitates adaptation to different learning scenarios than the one originally intended. 
However, the same LOs might actually score poorly in terms of its interactivity. The checklist, 
therefore, needs to be seen more as a minimum framework that should be used in a flexible way. 

In MELT the partners want to be able to provide access to learning content that meets nationally 
recognised quality criteria. However, it is also important to appreciate that some very high-quality LOs 
may meet the specific needs of a national curriculum but may not always have the ability to be used 


www.ejel.org 


129 


ISSN 1479-4403 


Electronic Journal of e- Learning Volume 7 Issue 2 2009, (127 - 136) 


as effectively (or maybe at all) by schools in other countries. For example, a text-heavy lesson plan in 
a minority European language may work splendidly in a national context but may simply be unusable 
by teachers in other countries. 

With this in mind during the content audit, MELT partners have begun to develop quality criteria that 
are defined in terms of the extent to which learning content has the potential to “travel well”; i.e. the 
extent to which LOs/assets can be easily used across national borders and in different curricula 
frameworks. At a commonsense level, some MELT content will obviously travel better than others. 
Learning assets such as pictures and sounds, for example, are obviously more reusable than a 
complex, Spanish language LO designed to convey facts about the Spanish War of Independence. 

Beyond this, an initial assumption in MELT is that content is more likely to “travel well” if it is: 

■ Modular: the parts of a content item are fully functional on their own. 

■ Adaptable: the LO can be modified, for instance from a configuration file, from a plain text file or 
because it is provided along with its source code or an authoring tool. 

Further discussions among partners also suggest that cross-border reuse of content will be more 
likely if LOs: 

■ Have a strong visual element and users can broadly understand what is the intended learning 
objective or topic (e.g. LOs may have little or no text; and include animations and simulations that 
are self-explanatory or have just a few text labels or icons/buttons for start, stop, etc.). 

■ Have been designed to be language customisable (“choose a language option”) and are already 
offered in more than one language. 

■ Address curriculum topics that could be considered trans-national (e.g. teaching “geometric 
shapes” or “the parts of the cell” are usually covered in every national curriculum but teaching the 
folklore of a very specific region is not). 

■ Are adaptable from a technical (e.g. LOs are supplied along with an authoring environment or 
tools) or IPR perspective (e.g. they are not made available under a “No derivatives” Creative 
Commons license which would prevent users from even translating the resource) (MELT 2007). 

1.5 Quality for Reuse project criteria 

A quality assurance strategy was implemented in “Quality for Reuse” (Q4R 2007) scientific project 
initiated by Tele-University of Quebec to improve effectiveness, efficiency and flexibility of LOs as well 
as proper storing and retrieval strategies. They have organised these strategies into four main groups, 
namely organisational strategies, and then three strategies inspired by the life-cycle of a LO, that is 
from its conception to its use /reuse (adaptations). Q4R quality assurance strategy is presented in 
Fig. 1: 




| Principle \ 

Ensure technical 
and organisational 
infrastructure 

T 



rganizatiom 

Strategies 


L 


A 


( 


V V 


Adopt Policy 
on Intellectual 
Property 




Ensure 

Management 

Support 


Figure 1 : Quality assurance strategy (Q4R 2007) 
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1.6 Lithuanian learning objects’ evaluation tool approved by the Ministry of 
Education and Science 

The last “Computer Teaching Aids Methodical and Technological Evaluation Criteria” for certification 
of educational software and content were approved in Lithuania in June 2008. 

These criteria are: 

■ Methodical aspects. 

■ User interface (incl. personalisation) (*suitable for technical evaluation - authors’ comment). 

■ LOs arrangement possibilities (*suitable for technical evaluation - authors’ comment). 

■ Communication and collaboration possibilities and tools (‘suitable for technical evaluation - 
authors’ comment). 

■ Technical features (incl. working stability) (‘suitable for technical evaluation - authors’ comment). 

■ Documentation. 

■ Implementation and maintenance expenditure (Computer... 2008). 

1.7 Conclusions of literature analysis and problems to solve 

It is obvious that all analysed LO technical evaluation tools have a number of limitations: 

■ LORI (Vargo et al. 2003), (Paulsson and Naeve 2006) and (MELT 2007) do not examine different 
LO life cycle stages. 

■ (Q4R 2007) insufficiently examines technical evaluation criteria before LO inclusion in the LO 
repository (LOR). 

■ All tools insufficiently examine LO reusability (incl. Interoperability) criteria. 

The approved Lithuanian set of evaluation criteria (see (Computer... 2008)) also has a number of 
limitations, e.g.: 

■ All LOs and services (e.g., LOs, LORs, VLEs) are evaluated against the same criteria. 

■ No metadata-related criteria are included. 

■ Approved technical evaluation criteria for e-content and activities do not reflect their reusability 
aspects overall (Kurilovas 2007, Kubilinskiene and Kurilovas 2008). 

Therefore this set of evaluation criteria is not suitable for technical quality evaluation of LOs and 
VLEs. It is obvious that more complex LO technical evaluation tool is needed. This tool should include 
LO technical evaluation criteria suitable for different LO life cycle stages, including criteria before, 
during and after LO inclusion in the repository as well as LO reusability criteria. LO reusability criteria 
should have the same weight as the other criteria. 

The authors’ research results show that Lithuanian education system needs a rapid growth of adapted 
LOs available for the teachers. It is obvious that this growth due to the limited financial and human 
resources is impossible without large scale adaptation, localisation, and reuse of LOs available in 
European Learning Resource Exchange system and other suitable repositories around the world. It is 
also clear that Lithuanian LO repositories should include a big number of teachers created LOs. 

Therefore Lithuanian education system needs high quality simple to use and clear enough LO 
technical evaluation tool based on scientific research in the area. 

1.8 Recommended learning objects technical evaluation tool 

The authors propose the original set of LO evaluation criteria based on flexible DLE model (i.e., 
reusability of DLE components, see (Dagiene and Kurilovas 2007)) as well as on conclusions of the 
analysis of LO technical evaluation criteria presented in Sections 1.1 - 1.6. This tool includes LO 
technical evaluation criteria suitable for different LO life cycle stages, including criteria before, during 
and after LO inclusion in the LOR, as well as LO reusability criteria. The tool combines (MELT 2007), 
(Paulsson and Naeve 2006), (Q4R 2007), (Vargo et al. 2003) and the authors’ own research results 
(e.g., (Kurilovas 2007)). The complex original LO technical quality evaluation tool is presented in Fig. 
2 . 
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Figure 2: Technical criteria for evaluation of LOs 

Additional LO evaluation criteria interconnected with technical criteria are: 

■ Licensing (clear rules, e.g. compliance with Creative Commons). 

■ Economic efficiency (taking into account the number of probable users in conformity with LO 
reusability level) (Kurilovas 2007) 

2. Technical evaluation of virtual learning environments 

The flexibility of DLE and learning personalisation possibilities for its users is achieved by separating 
‘content’ learning objects from ‘activity’ ones, separating LO metadata from LO repositories which can 
be on different servers, usage of highly adaptable open source tools to create/reuse LOs and provide 
online learning environments etc. (Dagiene and Kurilovas 2007, Kurilovas and Kubilinskiene 2008). 

The authors base their set of VLE evaluation criteria on flexible personalised approach to creation of 
DLE (Dagiene and Kurilovas 2007, Kurilovas and Kubilinskiene 2008) as well as mainly on two well- 
known VLE evaluation methods suitable for flexible personalised DLE: 

■ Methodology of technical evaluation of learning management systems (LMSs) (Technical... 
2004). 

■ Method of evaluation of open source e-learning platforms with the main focus is on adaptation 
issues (Graf and List 2005). 
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2.1 Methodology of technical evaluation of learning management systems 

Methodology of Technical Evaluation of LMSs (or VLEs) is a part of the Evaluation of Learning 
Management Software activity undertaken as part of the New Zealand Open Source VLE project 
(Technical... 2004). 

The evaluation criteria expand on a subset of the criteria, focusing on the technical aspects of VLEs: 

■ Overall architecture and implementation (*suitable for technical evaluation - authors’ comment): 
Scalability of the system; System modularity and extensibility; Possibility of multiple installations 
on a single platform; Reasonable performance optimisations; Look and feel is configurable; 
Security; Modular authentication; Robustness and stability; Installation, dependencies and 
portability. 

■ Interoperability (‘suitable for technical evaluation - authors’ comment): Integration is 
straightforward; LMS/VLE standards support. 

■ Cost of ownership. 

■ Strength of the development community: Installed base and longevity; Documentation; End-user 
community; Developer community; Open development process; Commercial support community. 

■ Licensing. 

■ Internationalisation and localisation (‘suitable for technical evaluation - authors’ comment): 
Localisable user interface; Localisation to relevant languages; Unicode text editing and storage; 
Time zones and date localisation; Alternative language support. 

■ Accessibility (‘suitable for technical evaluation - authors’ comment): Text-only navigation support; 
Scalable fonts and graphics. 

■ Document transformation (Technical... 2004, Kurilovas 2005). 

2.2 Virtual learning environments adaptation evaluation instrument 

(Graf and List 2005) paper presents an evaluation of open source e-learning platforms / VLEs with the 
main focus is on adaptation issues. 

Adaptation received very little coverage in e-learning platforms. An e-learning course should not be 
designed in a vacuum; rather, it should match students’ needs and desires as closely as possible, and 
adapt during course progression. The extended platform will be utilised in an operational teaching 
environment. Therefore, the overall functionality of the platform is as important as the adaptation 
capabilities, and the evaluation treats both issues. (Graf and List 2005) evaluation is also based on 
the qualitative weight and sum approach. After a pre-evaluation phase, nine platforms were analysed 
in detail. The detailed evaluation approach is focused on the adaptation category and its results. 

2.2. 1 Adaptation capabilities 

This section is focused on adaptability, personalisation, extensibility, and adaptivity capabilities of the 
platforms. (Graf and List 2005) research is focused on customisable adaptation only, which can be 
done without programming skills. 

■ Adaptability includes all facilities to customise the platform for the educational institution’s needs 
(e.g. the language or the design). 

■ Personalisation aspects indicate the facilities of each individual user to customise his/her own 
view of the platform. 

■ Extensibility is, in principle, possible for all open source products. Nevertheless, there can be big 
differences. For example, a good programming style or the availability of a documented 
application programming interfaces (API) are helpful. 

■ Adaptivity indicates all kinds of automatic adaptation to the individual user’s needs (e.g. personal 
annotations of learning objects or automatically adapted content). 

2.3 Conclusions of literature analysis and problems to solve 

Both analysed VLE technical evaluation tools have a number of limitations: 
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■ (Technical... 2004) tool practically does not examine adaptation capabilities criteria. 

■ (Graf and List 2005) tool insufficiently examines general technical criteria. 

Therefore more complex VLE technical evaluation tool is needed. It should include general technical 
evaluation criteria based on modular approach and interoperability, as well as adaptation capabilities 
criteria. 


2.4 Recommended virtual learning environments technical evaluation tool 

As it was assessed earlier, both analysed tools have a number of limitations. 

Therefore the authors propose the original complex set of VLE technical evaluation criteria combining 
general and adaptation criteria (see Fig. 3). 
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Figure 3: Technical criteria for evaluation of VLEs 


This tool includes general technical evaluation criteria based on modular approach and 
interoperability, as well as adaptation capabilities criteria. VLE adaptation capabilities criteria should 
have the same weight as the other criteria. 


2.5 Virtual learning environments experimental technical evaluation results 

The authors propose universal DLE components’ evaluation rating tool for evaluation of all main DLE 
components: LOs, their repositories and VLEs. It is clearer in comparison with (Technical... 2004) 
rating tool and more convenient in comparison with (Graf and List 2005) rating tool. This tool is based 
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on analysis of the level of feature’s support and the level of modification needed to reach the desired 
level of support (see Table 1 ). 


Table 1: Evaluation importance rating (Technical... 2006) 


Rating 

Explanation 


0 

Failed or feature does not exist 

1 

Has poor support and / or it can be done but with significant effort 

2 

Fair support but needs modification to reach the desired level of support 

3 

Good support and needs a minimal amount of effort 

4 

Excellent support and meets the criteria out of the box, minimal effort 


Each selected criterion is proposed to be given an importance rating to be used when evaluating LOs, 
repositories and VLEs. Major criteria have to be broken down into sub-criteria with each sub-criterion 
also having an importance rating. The importance rating range is 0-4, with 0 being the lowest and 4 
being of the highest importance. Each sub-criterion has then to be rated using a range of 0-4. The 
authors propose to weight each LO evaluation criteria equally and to use this simple and clear criteria 
rating system for evaluation of all components of DLE: LOs, LO repositories and VLEs. 

This universal DLE components’ evaluation rating tool was used by the authors (Kurilovas 2007) to 
evaluate three most popular open source VLEs against technical (both general and adaptation) 
criteria. The results of this evaluation are presented in Table 2. 


Table 2: VLE technical evaluation summary 


Technical evaluation criteria 

ATutor 

llias 

Moodle 


General criteria 

Architecture and implementation 

Rating 2 

Rating 1 

Rating 4 

Interoperability 

Rating 3 

Rating 3 

Rating 2 

Internationalisation and localisation 

Rating 1 

Rating 2 

Rating 3 

Accessibility 

Rating 4 

Rating 1 

Rating 2 

Interim evaluation rating: 

10 

7 

11 


Adaptation criteria 

Adaptability 

Rating 1 

Rating 2 

Rating 3 

Personalisation 

Rating 3 

Rating 3 

Rating 2 

Extensibility 

Rating 3 

Rating 4 

Rating 4 

Adaptivity 

Rating 1 

Rating 0 

Rating 1 

Interim evaluation rating: 

8 

9 

10 


Total evaluation rating: 

18 

16 

21 


In conformity with this practical evaluation results, Moodle is the best VLE from technical point of view. 


3. Conclusions 

3.1 Conclusions on learning objects technical evaluation 

The authors have analysed existing criteria for technical quality evaluation of LOs in Section 1. It was 
investigated that these criteria have a number of limitations, e.g., (1) LORI, (Paulsson and Naeve 
2006) and (MELT 2007) do not examine different LO life cycle stages, and (2) (Q4R 2007) 
insufficiently examines technical evaluation criteria before LO inclusion in the repository. All tools 
insufficiently examine LO reusability criteria. The approved Lithuanian set of evaluation criteria has 
many limitations, e.g. (1) in conformity with this tool all LOs and services (e.g., LOs, LORs, VLEs) 
have to be evaluated against the same criteria, (2) no metadata-related criteria are evaluated, and (3) 
these criteria do not reflect e-content and activities reusability aspects overall. Therefore more 
complex LO technical quality evaluation tool is needed. 

The authors propose an original more complex set of LOs technical quality evaluation criteria based 
on flexible DLE approach as well as on foreign LO technical evaluation criteria analysed in Sections 

1.2 - 1.5. These criteria were presented in Fig. 2. They are: (1) Before LO inclusion in the LOR: 
Narrow definition compliance; Reusability level: Interoperability, Pedagogical decontextualisation 
level, Cultural/learning diversity principles, and Accessibility; as well as Architecture; Working stability; 
Design and usability; (2) During LO inclusion in the LOR: Membership or Contribution Control 
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Strategies and Technical interoperability; (3) After LO inclusion in the LOR: Retrieval and Information 
quality. LO reusability (incl. Interoperability) criteria should have the same weight as the other criteria. 

3.2 Conclusions on virtual learning environments technical evaluation 

The authors have examined several VLE technical evaluation tools suitable for flexible DLE in 
Sections 2.1 - 2.2. It was investigated that these tools have a number of limitations, e.g. (1) 
(Technical... 2004) tool practically does not examine adaptation capabilities criteria, and (2) (Graf and 
List 2005) tool insufficiently examines general technical quality criteria. Therefore more complex VLE 
technical quality evaluation tool is needed. 

The authors propose an original more complex set of VLE technical evaluation criteria combining (1) 
General (Overall architecture and implementation; Interoperability; Internationalisation and 
localisation; Accessibility) and (2) Adaptation (Adaptability; Personalisation; Extensibility; Adaptivity) 
technical evaluation criteria (see Fig. 3). VLE adaptation capabilities criteria should have the same 
weight as the other criteria. 

The authors have also selected and propose to use the universal DLE components’ evaluation rating 
tool which is clearer and more convenient than investigated other foreign tools, and has evaluated 
three most popular open source VLEs against technical (both general and adaptation) criteria in 
conformity with this rating tool (see Table 2). 
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